MMPD: Multi-Domain Mobile Video Physiology Dataset.

Remote photoplethysmography (rPPG) is an attractive method for noninvasive, convenient and concomitant measurement of physiological vital signals. Public benchmark datasets have served a valuable role in the development of this technology and improvements in accuracy over recent years. However, there remain gaps in the public datasets. First, despite the ubiquity of cameras on mobile devices, there are few datasets recorded specifically with mobile phone cameras. Second, most datasets are relatively small and therefore are limited in diversity, both in appearance (e.g., skin tone), behaviors (e.g., motion) and environment (e.g., lighting conditions). In an effort to help the field advance, we present the Multi-domain Mobile Video Physiology Dataset (MMPD), comprising 11 hours of recordings from mobile phones of 33 subjects. The dataset is designed to capture videos with greater representation across skin tone, body motion, and lighting conditions. MMPD is comprehensive with eight descriptive labels and can be used in conjunction with the rPPG-toolbox [1]. The reliability of the dataset is verified by mainstream unsupervised methods and neural methods. The GitHub repository of our dataset: https://github.com/THU-CS-PI/MMPD_rPPG_dataset.

Video-based sympathetic arousal assessment via peripheral blood flow estimation.

Electrodermal activity (EDA) is considered a standard marker of sympathetic activity. However, traditional EDA measurement requires electrodes in steady contact with the skin. Can sympathetic arousal be measured using only an optical sensor, such as an RGB camera? This paper presents a novel approach to infer sympathetic arousal by measuring the peripheral blood flow on the face or hand optically. We contribute a self-recorded dataset of 21 participants, comprising synchronized videos of participants' faces and palms and gold-standard EDA and photoplethysmography (PPG) signals. Our results show that we can measure peripheral sympathetic responses that closely correlate with the ground truth EDA. We obtain median correlations of 0.57 to 0.63 between our inferred signals and the ground truth EDA using only videos of the participants' palms or foreheads or PPG signals from the foreheads or fingers. We also show that sympathetic arousal is best inferred from the forehead, finger, or palm.

Objective Communication Patterns Associated With Team Member Effectiveness in Real-World Virtual Teams.

OBJECTIVE: We explore the relationships between objective communication patterns displayed during virtual team meetings and established, qualitative measures of team member effectiveness. BACKGROUND: A key component of teamwork is communication. Automated measures of objective communication patterns are becoming more feasible and offer the ability to measure and monitor communication in a scalable, consistent and continuous manner. However, their validity in reflecting meaningful measures of teamwork processes are not well established, especially in real-world settings. METHOD: We studied real-world virtual student teams working on semester-long projects. We captured virtual team meetings using the Zoom video conferencing platform throughout the semester and periodic surveys comprising peer ratings of team member effectiveness. Leveraging audio transcripts, we examined relationships between objective measures of speaking time, silence gap duration and vocal turn-taking and peer ratings of team member effectiveness. RESULTS: Speaking time, speaking turn count, degree centrality and (marginally) speaking turn duration, but not silence gap duration, were positively related to individual-level team member effectiveness. Time in dyadic interactions and interaction count, but not interaction length, were positively related to dyad-level team member effectiveness. CONCLUSION: Our study highlights the relevance of objective measures of speaking time and vocal turn-taking to team member effectiveness in virtual project-based teams, supporting the validity of these objective measures and their use in future research. APPLICATION: Our approach offers a scalable, easy-to-use method for measuring communication patterns and team member effectiveness in virtual teams and opens the opportunity to study these patterns in a more continuous and dynamic manner.

Using High-Fidelity Avatars to Advance Camera-Based Cardiac Pulse Measurement.

Non-contact physiological measurement has the potential to provide low-cost, non-invasive health monitoring. However, machine vision approaches are often limited by the availability and diversity of annotated video datasets resulting in poor generalization to complex real-life conditions. To address these challenges, this work proposes the use of synthetic avatars that display facial blood flow changes and allow for systematic generation of samples under a wide variety of conditions. Our results show that training on both simulated and real video data can lead to performance gains under challenging conditions. We show strong performance on three large benchmark datasets and improved robustness to skin type and motion. These results highlight the promise of synthetic data for training camera-based pulse measurement; however, further research and validation is needed to establish whether synthetic data alone could be sufficient for training models.

Synthetic Data for Multi-Parameter Camera-Based Physiological Sensing.

Synthetic data is a powerful tool in training data hungry deep learning algorithms. However, to date, camera-based physiological sensing has not taken full advantage of these techniques. In this work, we leverage a high-fidelity synthetics pipeline for generating videos of faces with faithful blood flow and breathing patterns. We present systematic experiments showing how physiologically-grounded synthetic data can be used in training camera-based multi-parameter cardiopulmonary sensing. We provide empirical evidence that heart and breathing rate measurement accuracy increases with the number of synthetic avatars in the training set. Furthermore, training with avatars with darker skin types leads to better overall performance than training with avatars with lighter skin types. Finally, we discuss the opportunities that synthetics present in the domain of camera-based physiological sensing and limitations that need to be overcome.

Systematic analysis of video-based pulse measurement from compressed videos.

Camera-based physiological measurement enables vital signs to be captured unobtrusively without contact with the body. Remote, or imaging, photoplethysmography involves recovering peripheral blood flow from subtle variations in video pixel intensities. While the pulse signal might be easy to obtain from high quality uncompressed videos, the signal-to-noise ratio drops dramatically with video bitrate. Uncompressed videos incur large file storage and data transfer costs, making analysis, manipulation and sharing challenging. To help address these challenges, we use compression specific supervised models to mitigate the effect of temporal video compression on heart rate estimates. We perform a systematic evaluation of the performance of state-of-the-art algorithms across different levels, and formats, of compression. We demonstrate that networks trained on compressed videos consistently outperform other benchmark methods, both on stationary videos and videos with significant rigid head motions. By training on videos with the same, or higher compression factor than test videos, we achieve improvements in signal-to-noise ratio (SNR) of up to 3 dB and mean absolute error (MAE) of up to 6 beats per minute (BPM).

Non-contact imaging of peripheral hemodynamics during cognitive and psychological stressors.

Peripheral hemodynamics, measured via the blood volume pulse and vasomotion, provide a valuable way of monitoring physiological state. Camera imaging-based systems can be used to measure these peripheral signals without contact with the body, at distances of multiple meters. While researchers have paid attention to non-contact imaging photoplethysmography, the study of peripheral hemodynamics and the effect of autonomic nervous system activity on these signals has received less attention. Using a method, based on a tissue-like model of the skin, we extract melanin [Formula: see text] and hemoglobin [Formula: see text] concentrations from videos of the hand and face and show that significant decreases in peripheral pulse signal power (by 36% ± 29%) and vasomotion signal power (by 50% ± 26%) occur during periods of cognitive and psychological stress. Via three experiments we show that similar results are achieved across different stimuli and regions of skin (face and hand). While changes in peripheral pulse and vasomotion power were significant the changes in pulse rate variability were less consistent across subjects and tasks.

Simple and affordable imaging of multiple physiological parameters with RGB camera-based diffuse reflectance spectroscopy.

We propose a simple and affordable imaging technique to evaluate transcutaneously multiple physiological parameters by using a digital red-green-blue camera. In this method, the RGB-values were converted into tristimulus values in the CIE (Commission Internationale de l'Eclairage) XYZ color space, which is compatible with the common color spaces. Monte Carlo simulation for light transport in biological tissue was then performed to specify the relationship among the XYZ-values and the concentrations of oxygenated hemoglobin, deoxygenated hemoglobin, bilirubin, and melanin. The concentration of total hemoglobin and tissue oxygen saturation were also calculated from the estimated concentrations of oxygenated and deoxygenated hemoglobin. In vivo experiments with bile duct ligation in rats demonstrated that the estimated bilirubin concentration increased after ligation of the bile duct and reached around 22 mg/dl at 116 h after the onset of ligation, which corresponds to the ground truth value of bilirubin measured by a commercially available transcutaneous bilirubinometer. Experiments with rats while varying the fraction of inspired oxygen demonstrated that oxygenated hemoglobin and deoxygenated hemoglobin decreased and increased, respectively, as the fraction of inspired oxygen decreased. Consequently, tissue oxygen saturation dramatically decreased. We further extended the method to a non-contact imaging photo-plethysmograph and estimation of the percutaneous oxygen saturation. An empirical formula to estimate percutaneous oxygen saturation was derived from the pulse wave amplitudes of oxygenated and deoxygenated hemoglobin. The estimated percutaneous oxygen saturation dropped remarkably when a faction of inspired oxygen was below 19%, indicating the onset of hypoxemia due to hypoxia, whereas the tissue oxygen saturation decreased gradually according to the reduction of the faction of inspired oxygen. The results in this study indicate the potential of this method for imaging of multiple physiological parameters in skin tissue and evaluating an optical biomedical imaging technique that enables cost-effective, easy-to-use, portable, remotely administered, and/or point-of-care solutions.

Wearable Motion-Based Heart Rate at Rest: A Workplace Evaluation.

This paper studies the feasibility of using low-cost motion sensors to provide opportunistic heart rate assessments from ballistocardiographic signals during restful periods of daily life. Three wearable devices were used to capture peripheral motions at specific body locations (head, wrist, and trouser pocket) of 15 participants during five regular workdays each. Three methods were implemented to extract heart rate from motion data and their performance was compared to those obtained with an FDA-cleared device. With a total of 1358 h of naturalistic sensor data, our results show that providing accurate heart rate estimations from peripheral motion signals is possible during relatively "still" moments. In our real-life workplace study, the head-mounted device yielded the most frequent assessments (22.98% of the time under 5 beats per minute of error) followed by the smartphone in the pocket (5.02%) and the wrist-worn device (3.48%). Most importantly, accurate assessments were automatically detected by using a custom threshold based on the device jerk. Due to the pervasiveness and low cost of wearable motion sensors, this paper demonstrates the feasibility of providing opportunistic large-scale low-cost samples of resting heart rate.

iPhys: An Open Non-Contact Imaging-Based Physiological Measurement Toolbox.

Imaging-based, non-contact measurement of physiology (including imaging photoplethysmography and imaging ballistocardiography) is a growing field of research. There are several strengths of imaging methods that make them attractive. They remove the need for uncomfortable contact sensors and can enable spatial and concomitant measurement from a single sensor. Furthermore, cameras are ubiquitous and often low-cost solutions for sensing. Open source toolboxes help accelerate the progress of research by providing a means to compare new approaches against standard implementations of the state-of-the-art. We present an open source imaging-based physiological measurement toolbox with implementations of many of the most frequently employed computational methods. We hope that this toolbox will contribute to the advancement of noncontact physiological sensing methods. Code: https://github.com/danmcduff/iphys-toolbox.

Using Non-Contact Imaging Photoplethysmography to Recover Diurnal Patterns in Heart Rate.

Daily patterns in cardiovascular signals can reveal important information about physiological processes, health and well-being. Traditionally, contact sensors have been used to collect longitudinal data of this kind. However, recent advances in non-contact imaging techniques have led to algorithms that can be used to measure vital signs unobtrusively. Imaging methods are highly scalable due to the availability of webcams and computing devices making them attractive for longitudinal, in-situ measurement. Using a software tool we captured over 1,000 hours of non-contact heart rate measurements, via imaging photoplethysmography. Using these data we were able to recover diurnal patterns in heart rate during the working day. Non-contact sensing techniques hold much promise but also raise ethical issues that need to be addressed seriously within the biomedical engineering community.

Spectral Estimation Methods for Evaluating iPPG Pulse Rate Variability.

Non-contact measurement of physiological parameters, like pulse rate variability (PRV), has numerous applications in medicine and affective computing. PRV is an informative measure of autonomic nervous system activity. Spectral estimation from unevenly sampled, non-stationary data is integral to pulse rate variability frequency-domain analysis. We present the first comparison of results of PRV computation using the Lomb-Scargle method and Bayesian Spectral Estimation. The Lomb-Scargle method performs well, even in the presence of missing beats. However, the Bayesian Spectral Estimation method has advantages when tracking changes in amplitude and frequency. We illustrate these characteristics with results from synthetic data and real non-contact imaging photoplethysmography measurements.

InPhysible: Camouflage Against Video-Based Physiological Measurement.

Imaging photoplethysmography (iPPG) is a powerful set of methods for measuring physiological signals from video. Recent advances have shown that a low-cost webcam can be used to measure heart rate, blood flow, respiration, blood oxygen levels and stress. While these methods have many beneficial applications, the unobtrusive and ubiquitous nature of the sensors risk exposing people to unwanted measurement. We present InPhysible the first camouflage system against video- based physiological measurement. The infra-red system can be embedded into any pair of glasses, or other headwear, and disrupts the measurement of the iPPG signal while being imperceptible by the human eye. Our system is flexible and can simulate realistic pulse signals to hinder heart rate measurement. In this paper we present the design of our prototype and a user study validating its efficacy. Finally, we discuss the limitations and implications for data privacy and security.

Fusing Partial Camera Signals for Noncontact Pulse Rate Variability Measurement.

Remote camera-based measurement of physiology has great potential for healthcare and affective computing. Recent advances in computer vision and signal processing have enabled photoplethysmography (PPG) measurement using commercially available cameras. However, there remain challenges in recovering accurate noncontact PPG measurements in the presence of rigid head motion. When a subject is moving, their face may be turned away from one camera, be obscured by an object, or move out of the frame resulting in missing observations. As the calculation of pulse rate variability (PRV) requires analysis over a time window of several minutes, the effect of missing observations on such features is deleterious. We present an approach for fusing partial color-channel signals from an array of cameras that enable physiology measurements to be made from moving subjects, even if they leave the frame of one or more cameras, which would not otherwise be possible with only a single camera. We systematically test our method on subjects ( N=25) using a set of six, 5-min tasks (each repeated twice) involving different levels of head motion. This results in validation across 25 h of measurement. We evaluate pulse rate and PRV parameter estimation including statistical, geometric, and frequency-based measures. The median absolute error in pulse rate measurements was 0.57 beats-per-minute (BPM). In all but two tasks with the greatest motion, the median error was within 0.4 BPM of that from a contact PPG device. PRV estimates were significantly improved using our proposed approach compared to an alternative not designed to handle missing values and multiple camera signals; the error was reduced by over 50%. Without our proposed method, errors in pulse rate would be very high, and estimation of PRV parameters would not be feasible due to significant data loss.

A large-scale analysis of sex differences in facial expressions.

There exists a stereotype that women are more expressive than men; however, research has almost exclusively focused on a single facial behavior, smiling. A large-scale study examines whether women are consistently more expressive than men or whether the effects are dependent on the emotion expressed. Studies of gender differences in expressivity have been somewhat restricted to data collected in lab settings or which required labor-intensive manual coding. In the present study, we analyze gender differences in facial behaviors as over 2,000 viewers watch a set of video advertisements in their home environments. The facial responses were recorded using participants' own webcams. Using a new automated facial coding technology we coded facial activity. We find that women are not universally more expressive across all facial actions. Nor are they more expressive in all positive valence actions and less expressive in all negative valence actions. It appears that generally women express actions more frequently than men, and in particular express more positive valence actions. However, expressiveness is not greater in women for all negative valence actions and is dependent on the discrete emotional state.

A survey of remote optical photoplethysmographic imaging methods.

In recent years researchers have presented a number of new methods for recovering physiological parameters using just low-cost digital cameras and image processing. The ubiquity of digital cameras presents the possibility for many new, low-cost applications of vital sign monitoring. In this paper we present a review of the work on remote photoplethysmographic (PPG) imaging using digital cameras. This review specifically focuses on the state-of-the-art in PPG imaging where: 1) measures beyond pulse rate are evaluated, 2) non-ideal conditions (e.g., the presence of motion artifacts) are explored, and 3) use cases in relevant environments are demonstrated. We discuss gaps within the literature and future challenges for the research community. To aid in the continuing advancement of PPG imaging research, we are making available a website with the references collected for this review as well as information on available code and datasets of interest. It is our hope that this website will become a valuable resource for the PPG imaging community. The site can be found at: http://web.mit.edu/~djmcduff/www/ remote-physiology.html.

Biophone: Physiology monitoring from peripheral smartphone motions.

The large-scale adoption of smartphones during recent years has created many opportunities to improve health monitoring and care delivery. In this work, we demonstrate that motion sensors available in off-the-shelf smartphones can capture physiological parameters of a person during stationary postures, even while being carried in a bag or a pocket. In particular, we develop methods to extract heart and breathing rates from accelerometer data and compare them with measurements obtained with FDA-cleared sensors. We evaluated their accuracy on 12 people across different still body postures (pre- and post- exercise) and were able to reach mean absolute errors of 1.16 beats per minute (STD: 3) and 0.26 breaths per minute (STD: 0.5) when considering different conditions. Furthermore, we evaluated the same methods during regular phone activities, such as when watching a video or listening to a conversation, yielding increased but still comparable error rates for some conditions.

Remote detection of photoplethysmographic systolic and diastolic peaks using a digital camera.

We present a new method for measuring photoplethysmogram signals remotely using ambient light and a digital camera that allows for accurate recovery of the waveform morphology (from a distance of 3 m). In particular, we show that the peak-to-peak time between the systolic peak and diastolic peak/inflection can be automatically recovered using the second-order derivative of the remotely measured waveform. We compare measurements from the face with those captured using a contact fingertip sensor and show high agreement in peak and interval timings. Furthermore, we show that results can be significantly improved using orange, green, and cyan color channels compared to the tradition red, green, and blue channel combination. The absolute error in interbeat intervals was 26 ms and the absolute error in mean systolic-diastolic peak-to-peak times was 12 ms. The mean systolic-diastolic peak-to-peak times measured using the contact sensor and the camera were highly correlated, ρ = 0.94 (p 0.001). The results were obtained with a camera frame-rate of only 30 Hz. This technology has significant potential for advancing healthcare.

Improvements in remote cardiopulmonary measurement using a five band digital camera.

Remote measurement of the blood volume pulse via photoplethysmography (PPG) using digital cameras and ambient light has great potential for healthcare and affective computing. However, traditional RGB cameras have limited frequency resolution. We present results of PPG measurements from a novel five band camera and show that alternate frequency bands, in particular an orange band, allowed physiological measurements much more highly correlated with an FDA approved contact PPG sensor. In a study with participants (n = 10) at rest and under stress, correlations of over 0.92 (p 0.01) were obtained for heart rate, breathing rate, and heart rate variability measurements. In addition, the remotely measured heart rate variability spectrograms closely matched those from the contact approach. The best results were obtained using a combination of cyan, green, and orange (CGO) bands; incorporating red and blue channel observations did not improve performance. In short, RGB is not optimal for this problem: CGO is better. Incorporating alternative color channel sensors should not increase the cost of such cameras dramatically.

Remote measurement of cognitive stress via heart rate variability.

Remote detection of cognitive load has many powerful applications, such as measuring stress in the workplace. Cognitive tasks have an impact on breathing and heart rate variability (HRV). We show that changes in physiological parameters during cognitive stress can be captured remotely (at a distance of 3m) using a digital camera. A study (n=10) was conducted with participants at rest and under cognitive stress. A novel five band digital camera was used to capture videos of the face of the participant. Significantly higher normalized low frequency HRV components and breathing rates were measured in the stress condition when compared to the rest condition. Heart rates were not significantly different between the two conditions. We built a person-independent classifier to predict cognitive stress based on the remotely detected physiological parameters (heart rate, breathing rate and heart rate variability). The accuracy of the model was 85% (35% greater than chance).